System and method for predicting behaviors of detected objects

ABSTRACT

Aspects of the invention relate generally to autonomous vehicles. Specifically, the features described may be used alone or in combination in order to improve the safety, use, driver experience, and performance of these vehicles.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit of the filing dates of U.S.Provisional Application No. 61/390,094, entitled “AUTONOMOUS VEHICLES,”filed Oct. 5, 2010, and U.S. Provisional Application No. 61/391,271,entitled “AUTONOMOUS VEHICLES,” filed Oct. 8, 2010, the entiredisclosures of which are hereby incorporated herein by reference.

BACKGROUND

Autonomous vehicles use various computing systems to aid in thetransport passengers from one location to another. Some autonomousvehicles may require some initial input or continuous input from anoperator, such as a pilot, driver, or passenger. Other systems, forexample autopilot systems, may be used only when the system has beenengaged, which permits the operator to switch from a manual mode (wherethe operator exercises a high degree of control over the movement of thevehicle) to an autonomous mode (where the vehicle essentially drivesitself) to modes that lie somewhere in between.

BRIEF SUMMARY

In various aspects, the invention provides a vehicle having a steeringdevice (e.g., wheels that turn in the case of an automobile and a rudderin the case of a boat) and engine. The steering device may be controlledby a first user input controller (e.g., a steering wheel in the cockpitof a car), the engine may be controlled by a second user inputcontroller (e.g., accelerator in the case of a car or a throttle in thecase of boat), and both the engine and device may be controlled by aprocessor capable of executing computer instructions. The vehicleinclude one or more sensors (e.g., cameras, radar, laser range finders)for capturing information relating to the environment in which thevehicle is operating. The processor receives data from the sensors and,based in part on data from the sensors or received from external sourcesor both, issues a navigation command, where a navigations commandcomprises a command to the steering device relating to the intendeddirection of the vehicle (e.g., a command to turn the front wheels of acar 10 degrees to the left) or to the engine relating to the intendedvelocity of the vehicle (e.g., a command to accelerate). Navigationcommands may also include commands to brakes to slow the vehicle down,as well as other commands affecting the movement of the vehicle.

In one aspect, sensors are used to detect an object external to thevehicle, and data corresponding to the object is sent to a processor.The processor analyzes the data corresponding to the object to determinethe classification and the state of the object. The processor thenpredicts the likely behavior of the object by accessing behavior datafor entities having a classification and state similar to the object.The vehicle may then orient itself in an intended position and velocitybased at least in part on the likely behavior of the object.

In another aspect, the classification of the object external to thevehicle includes classifying the object as an automobile, a pedestrian,or a bicycle. In addition, the object may be further classified such asby determining the type of automobile, or by classifying the objectbased on a logo, bumper sticker, or license plate.

In yet another aspect, the state of the object external to the vehiclerelates to at least one of: location, traffic lane in which the objectis traveling, speed, acceleration, entry onto a road, exit off of aroad, activation of headlights, activation of taillights, or activationof blinkers.

In still another aspect, the behavior data is collected by tracking thestates of numerous entities at one or more locations. This tracking maybe performed by satellite imagery, roadside cameras, on-board GPS data,or via sensor data acquired from other nearby entities.

In another aspect, the command to orient the vehicle may includepositioning the vehicle at a predetermined distance from the object, thepredetermined distance being based at least in part on theclassification of the object. In addition, the likely behavior of theobject may be provided as a probability of the object entering to one ormore states.

In still another aspect, the vehicle receives updated behavior data, forexample, from a remote server. The vehicle may then be autonomouslycontrolled based, at least in part, on the updated behavior data.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a functional diagram of a system in accordance with an aspectof the invention.

FIG. 2 is an exemplary design of the interior of an autonomous vehiclein accordance with an aspect of the invention.

FIG. 3 is a view of the exterior of an exemplary vehicle in accordancewith an aspect of the invention.

FIGS. 4A-4D are views of the sensor fields for an autonomous vehicle.

FIGS. 5A and 5B are views of an autonomous vehicle in proximity to anexternal object.

FIG. 6 is a flow diagram in accordance with an aspect of the invention.

FIG. 7 is a function diagram of a system in accordance with an aspect ofthe invention.

DETAILED DESCRIPTION

Aspects of the disclosure relate generally to an autonomous drivingsystem. In particular, a vehicle implementing the autonomous drivingsystem is capable of detecting and reacting to surrounding objects. Someof the detected objects will be mobile, such as pedestrians,automobiles, and bicycles. As set forth below, the autonomous drivingsystem is able to identify and classify nearby objects. In addition,based on the object's classification, the autonomous driving system maypredict the object's likely movements and behavior. In turn, the vehiclemay react to nearby objects in a way that decreases the likelihood of anaccident and increases the efficiency of travel.

As shown in FIG. 1, an autonomous driving system 100 in accordance withone aspect of the invention includes a vehicle 101 with variouscomponents. While certain aspects of the invention are particularlyuseful in connection with specific types of vehicles, the vehicle may beany type of vehicle including, but not limited to, cars, trucks,motorcycles, busses, boats, airplanes, helicopters, lawnmowers,recreational vehicles, amusement park vehicles, trams, golf carts,trains, and trolleys. The vehicle may have one or more computers, suchas computer 110 containing a processor 120, memory 130 and othercomponents typically present in general purpose computers.

The memory 130 stores information accessible by processor 120, includinginstructions 132 and data 134 that may be executed or otherwise used bythe processor 120. The memory 130 may be of any type capable of storinginformation accessible by the processor, including a computer-readablemedium, or other medium that stores data that may be read with the aidof an electronic device, such as a hard-drive, memory card, ROM, RAM,DVD or other optical disks, as well as other write-capable and read-onlymemories. Systems and methods may include different combinations of theforegoing, whereby different portions of the instructions and data arestored on different types of media.

The instructions 132 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor. For example, the instructions may be stored as computer codeon the computer-readable medium. In that regard, the terms“instructions” and “programs” may be used interchangeably herein. Theinstructions may be stored in object code format for direct processingby the processor, or in any other computer language including scripts orcollections of independent source code modules that are interpreted ondemand or compiled in advance. Functions, methods and routines of theinstructions are explained in more detail below.

The data 134 may be retrieved, stored or modified by processor 120 inaccordance with the instructions 132. For instance, although the systemand method is not limited by any particular data structure, the data maybe stored in computer registers, in a relational database as a tablehaving a plurality of different fields and records, XML documents orflat files. The data may also be formatted in any computer-readableformat. By further way of example only, image data may be stored asbitmaps comprised of grids of pixels that are stored in accordance withformats that are compressed or uncompressed, lossless (e.g., BMP) orlossy (e.g., JPEG), and bitmap or vector-based (e.g., SVG), as well ascomputer instructions for drawing graphics. The data may comprise anyinformation sufficient to identify the relevant information, such asnumbers, descriptive text, proprietary codes, references to data storedin other areas of the same memory or different memories (including othernetwork locations) or information that is used by a function tocalculate the relevant data.

The processor 120 may be any conventional processor, such as processorsfrom Intel Corporation or Advanced Micro Devices. Alternatively, theprocessor may be a dedicated device such as an ASIC. Although FIG. 1functionally illustrates the processor, memory, and other elements ofcomputer 110 as being within the same block, it will be understood bythose of ordinary skill in the art that the processor and memory mayactually comprise multiple processors and memories that may or may notbe stored within the same physical housing. For example, memory may be ahard drive or other storage media located in a housing different fromthat of computer 110. Accordingly, references to a processor or computerwill be understood to include references to a collection of processorsor computers or memories that may or may not operate in parallel. Ratherthan using a single processor to perform the steps described herein someof the components such as steering components and decelerationcomponents may each have their own processor that only performscalculations related to the component's specific function.

In various of the aspects described herein, the processor may be locatedremote from the vehicle and communicate with the vehicle wirelessly. Inother aspects, some of the processes described herein are executed on aprocessor disposed within the vehicle and others by a remote processor,including taking the steps necessary to execute a single maneuver.

Computer 110 may include all of the components normally used inconnection with a computer, such as a central processing unit (CPU),memory (e.g., RAM and internal hard drives) storing data 134 andinstructions such as a web browser, an electronic display 142 (e.g., amonitor having a screen, a small LCD touch-screen or any otherelectrical device that is operable to display information), user input(e.g., a mouse, keyboard, touch screen and/or microphone), as well asvarious sensors (e.g. a video camera) for gathering the explicit (e.g. agesture) or implicit (e.g. “the person is asleep”) information about thestates and desires of a person.

The vehicle may also include a geographic position component 144 incommunication with computer 110 for determining the geographic locationof the device. For example, the position component may include a GPSreceiver to determine the device's latitude, longitude and/or altitudeposition. Other location systems such as laser-based localizationsystems, inertial-aided GPS, or camera-based localization may also beused to identify the location of the vehicle. The location of thevehicle may include an absolute geographical location, such as latitude,longitude, and altitude as well as relative location information, suchas location relative to other cars immediately around it which can oftenbe determined with less noise than absolute geographical location.

The device may also include other features in communication withcomputer 110, such as an accelerometer, gyroscope or anotherdirection/speed detection device 146 to determine the direction andspeed of the vehicle or changes thereto. By way of example only, device146 may determine its pitch, yaw or roll (or changes thereto) relativeto the direction of gravity or a plane perpendicular thereto. The devicemay also track increases or decreases in speed and the direction of suchchanges. The device's provision of location and orientation data as setforth herein may be provided automatically to the user, computer 110,other computers and combinations of the foregoing.

The computer 110 may control the direction and speed of the vehicle bycontrolling various components. By way of example, if the vehicle isoperating in a completely autonomous mode, computer 110 may cause thevehicle to accelerate (e.g., by increasing fuel or other energy providedto the engine), decelerate (e.g., by decreasing the fuel supplied to theengine or by applying brakes) and change direction (e.g., by turning thefront two wheels).

Computer 110 may also control status indicators 138, in order to conveythe status of the vehicle and its components to a passenger of vehicle101. For example, vehicle 101 may be equipped with a display 225 fordisplaying information relating to the overall status of the vehicle,particular sensors, or computer 110 in particular. The display 225 mayinclude computer generated images of the vehicle's surroundingsincluding, for example, the status of the computer (cruise), the vehicleitself 410, roadways 420, intersections 430, as well as other objectsand information.

Computer 110 may use visual or audible cues to indicate whether computer110 is obtaining valid data from the various sensors, whether thecomputer is partially or completely controlling the direction or speedof the car or both, whether there are any errors, etc. Vehicle 101 mayalso include a status indicating apparatus, such as status bar 230, toindicate the current status of vehicle 101. In the example of FIG. 2,status bar 230 displays “D” and “2 mph” indicating that the vehicle ispresently in drive mode and is moving at 2 miles per hour. In thatregard, the vehicle may display text on an electronic display,illuminate portions of vehicle 101, or provide various other types ofindications. In addition, the computer may also have external indicatorswhich indicate whether, at the moment, a human or an automated system isin control of the vehicle, that are readable by humans, other computers,or both.

In one example, computer 110 may be an autonomous driving computingsystem capable of communicating with various components of the vehicle.For example, computer 110 may be in communication with the vehicle'sconventional central processor 160 and may send and receive informationfrom the various systems of vehicle 101, for example the braking 180,acceleration 182, signaling 184, and navigation 186 systems in order tocontrol the movement, speed, etc. of vehicle 101. In addition, whenengaged, computer 110 may control some or all of these functions ofvehicle 101 and thus be fully or partially autonomous. It will beunderstood that although various systems and computer 110 are shownwithin vehicle 101, these elements may be external to vehicle 101 orphysically separated by large distances.

FIG. 2 depicts an exemplary design of the interior of an autonomousvehicle. The autonomous vehicle may include all of the features of anon-autonomous vehicle, for example: a steering apparatus, such assteering wheel 210; a navigation display apparatus, such as navigationdisplay 215; and a gear selector apparatus, such as gear shifter 220.

The vehicle may include components for detecting objects external to thevehicle such as other vehicles, obstacles in the roadway, trafficsignals, signs, trees, etc. The detection system may include lasers,sonar, radar, cameras or any other detection devices. For example, ifthe vehicle is a small passenger car, the car may include a lasermounted on the roof or other convenient location. In one aspect, thelaser may measure the distance between the vehicle and the objectsurfaces facing the vehicle by spinning on its axis and changing itspitch. The vehicle may also include various radar detection units, suchas those used for adaptive cruise control systems. The radar detectionunits may be located on the front and back of the car as well as oneither side of the front bumper. In another example, a variety ofcameras may be mounted on the car at distances from one another whichare known so that the parallax from the different images may be used tocompute the distance to various objects which are captured by 2 or morecameras. These sensors allow the vehicle to understand and potentiallyrespond to its environment in order to maximize safety for passengers aswell as objects or people in the environment.

In addition to the sensors described above, the computer may also useinput from sensors typical non-autonomous vehicles. For example, thesesensors may include tire pressure sensors, engine temperature sensors,brake heat sensors, break pad status, tire tread sensors, fuel sensors,oil level and quality sensors, air quality sensors (for detectingtemperature, humidity, or particulates in the air), etc.

Many of these sensors provide data that is processed by the computer inreal-time, that is, the sensors may continuously update their output toreflect the environment being sensed at or over a range of time, andcontinuously or as-demanded provide that updated output to the computerso that the computer can determine whether the vehicle's then-currentdirection or speed should be modified in response to the sensedenvironment.

FIG. 3 illustrates a particular embodiment for a small passenger vehicle301 that includes lasers 310 and 311, mounted on the front and top ofthe vehicle, respectively. Laser 310 may have a range of approximately150 meters, a thirty degree vertical field of view, and approximately athirty degree horizontal field of view. Laser 311 may have a range ofapproximately 50-80 meters, a thirty degree vertical field of view, anda 360 degree horizontal field of view. The lasers may provide thevehicle with range and intensity information which the computer may useto identify the location and distance of various objects. In one aspect,the lasers may measure the distance between the vehicle and the objectsurfaces facing the vehicle by spinning on its axis and changing itspitch.

The vehicle may also include various radar detection units, such asthose used for adaptive cruise control systems. The radar detectionunits may be located on the front and back of the car as well as oneither side of the front bumper. As shown in the example of FIG. 3,vehicle 301 includes radar detection units 320-323 located on the side(only one side being shown), front and rear of the vehicle. Each ofthese radar detection units may have a range of approximately 200 metersfor an approximately 18 degree field of view as well as a range ofapproximately 60 meters for an approximately 56 degree field of view.

In another example, a variety of cameras may be mounted on the vehicle.The cameras may be mounted at predetermined distances so that theparallax from the images of 2 or more cameras may be used to compute thedistance to various objects. As shown in FIG. 3, vehicle 301 may include2 cameras 330-331 mounted under a windshield 340 near the rear viewmirror (not shown). Camera 330 may include a range of approximately 200meters and an approximately 30 degree horizontal field of view, whilecamera 331 may include a range of approximately 100 meters and anapproximately 60 degree horizontal field of view.

Each sensor may be associated with a particular sensor field in whichthe sensor may be used to detect objects. FIG. 4A is a top-down view ofthe approximate sensor fields of the various sensors. FIG. 4B depictsthe approximate sensor fields 410 and 411 for lasers 310 and 311,respectively based on the fields of view for these sensors. For example,sensor field 410 includes an approximately 30 degree horizontal field ofview for approximately 150 meters, and sensor field 411 includes a 360degree horizontal field of view for approximately 80 meters.

FIG. 4D depicts the approximate sensor fields 420A-423B and for radardetection units 320-323, respectively, based on the fields of view forthese sensors. For example, radar detection unit 320 includes sensorfields 420A and 420B. Sensor field 420A includes an approximately 18degree horizontal field of view for approximately 200 meters, and sensorfield 420B includes an approximately 56 degree horizontal field of viewfor approximately 80 meters. Similarly, radar detection units 321-323include sensor fields 421A-423A and 421B-423B. Sensor fields 421A-423Ainclude an approximately 18 degree horizontal field of view forapproximately 200 meters, and sensor fields 421B-423B include anapproximately 56 degree horizontal field of view for approximately 80meters. Sensor fields 421A and 422A extend passed the edge of FIGS. 4Aand 4D.

FIG. 4C depicts the approximate sensor fields 430-431 cameras 330-331,respectively, based on the fields of view for these sensors. Forexample, sensor field 430 of camera 330 includes a field of view ofapproximately 30 degrees for approximately 200 meters, and sensor field431 of camera 430 includes a field of view of approximately 60 degreesfor approximately 100 meters.

In another example, an autonomous vehicle may include sonar devices,stereo cameras, a localization camera, a laser, and a radar detectionunit each with different fields of view. The sonar may have a horizontalfield of view of approximately 60 degrees for a maximum distance ofapproximately 6 meters. The stereo cameras may have an overlappingregion with a horizontal field of view of approximately 50 degrees, avertical field of view of approximately 10 degrees, and a maximumdistance of approximately 30 meters. The localization camera may have ahorizontal field of view of approximately 75 degrees, a vertical fieldof view of approximately 90 degrees and a maximum distance ofapproximately 10 meters. The laser may have a horizontal field of viewof approximately 360 degrees, a vertical field of view of approximately30 degrees, and a maximum distance of 100 meters. The radar may have ahorizontal field of view of 60 degrees for the near beam, 30 degrees forthe far beam, and a maximum distance of 200 meters.

The sensors described may be used to identify, track and predict themovements of pedestrians, bicycles, other vehicles, or objects in theroadway. For example, the sensors may provide the location and shapeinformation of objects surrounding the vehicle to computer 110, which inturn may identify the object as another vehicle. The object's currentmovement may be also be determined by the sensor (e.g., the component isa self-contained speed radar detector) or by the computer 110 based oninformation provided by the sensors (e.g., by comparing changes in theobject's position data over time).

The computer may change the vehicle's current path and speed based onthe presence of detected objects. For example, the vehicle mayautomatically slow down if its current speed is 50 mph and it detects,by using its cameras and using optical-character recognition, that itwill shortly pass a sign indicating that the speed limit is 35 mph. Yetfurther, if the computer determines that an object is obstructing theintended path of the vehicle, it may maneuver the vehicle around theobstruction.

Yet further, the vehicle's computer system may predict a detectedobject's expected movement. In one aspect, the computer system 110 maysimply predict the object's future movement based solely on the object'sinstant direction, acceleration/deceleration and velocity, e.g., thatthe object's current direction and movement will continue.

Once an object is detected, the system may determine the type of theobject, for example, a traffic cone, person, car, truck or bicycle, anduse this information to predict the object's future behavior. Objectsmay be identified by using an object classifier 148 which may considervarious characteristics of the detected objects, such as the size of anobject (bicycles are larger than a breadbox and smaller than a car), thespeed of the object (bicycles do not tend to go faster than 40 miles perhour or slower than 0.1 miles per hour), the heat coming from thebicycle (bicycles tend to have rider that emit heat from their bodies),etc. In addition, the object may be classified based on specificattributes of the object, such as information contained on a licenseplate, bumper sticker, or logos that appear on the vehicle.

In some examples, objects identified by the vehicle may not actuallyrequire the vehicle to alter its course. For example, if there were asand storm, the vehicle may detect the sand as one or many objects, butneed not alter its trajectory, though it may slow or stop itself forsafety reasons.

In another example, the scene external to the vehicle need not besegmented from input of the various sensors and nor do objects need tobe classified for the vehicle to take a responsive action. Rather thevehicle may take one or more actions based on the color and/or shape ofan object.

The system may also rely on information that is independent of thedetected object's movement to predict the object's next action. By wayof example, if the vehicle determines that another object is a bicyclethat is beginning to ascend a steep hill in front of the vehicle, thecomputer may predict that the bicycle will soon slow down—and will slowthe vehicle down accordingly—regardless of whether the bicycle iscurrently traveling at a somewhat high speed.

It will be understood that the forgoing methods of identifying,classifying, and reacting to objects external to the vehicle may be usedalone or in any combination in order to increase the likelihood ofavoiding a collision.

By way of further example, the system may determine that an object nearthe vehicle is another car in a turn-only lane (e.g., by analyzing imagedata that captures the other car, the lane the other car is in, and apainted left-turn arrow in the lane). In that regard, the system maypredict that the other car may turn at the next intersection.

The computer may cause the vehicle to take particular actions inresponse to the predicted actions of the surrounding objects. Forexample, if the computer 110 determines that the other car is turning atthe next intersection as noted above, the computer may slow the vehicledown as it approaches the intersection. In this regard, the predictedbehavior of other objects is based not only on the type of object andits current trajectory, but also based on some likelihood that theobject may obey traffic rules or pre-determined behaviors. In anotherexample, the system may include a library of rules about what objectswill do in various situations. For example, a car in a left-most lanethat has a left-turn arrow mounted on the light will very likely turnleft when the arrow turns green. The library may be built manually, orby the vehicle's observation of other vehicles (autonomous or not) onthe roadway. The library may begin as a human built set of rules whichmay be improved by the vehicle's observations. Similarly, the librarymay begin as rules learned from vehicle observation and have humansexamine the rules and improve them manually. This observation andlearning may be accomplished by, for example, tools and techniques ofmachine learning. The rules library may be included in computer 110 ormay alternatively be accessible to the vehicle 101 via a remote server,such as server 710 of FIG. 7.

In addition to processing data provided by the various sensors, thecomputer may rely on environmental data that was obtained at a previouspoint in time and is expected to persist regardless of the vehicle'spresence in the environment. For example, data 134 may include detailedmap information 136, e.g., highly detailed maps identifying the shapeand elevation of roadways, lane lines, intersections, crosswalks, speedlimits, traffic signals, buildings, signs, real time trafficinformation, or other such objects and information. For example, the mapinformation may include explicit speed limit information associated withvarious roadway segments. The speed limit data may be entered manuallyor scanned from previously taken images of a speed limit sign using, forexample, optical-character recognition. The map information may includethree-dimensional terrain maps incorporating one or more of objectslisted above. For example, the vehicle may determine that another car isexpected to turn based on real-time data (e.g., using its sensors todetermine the current GPS position of another car) and other data (e.g.,comparing the GPS position with previously-stored lane-specific map datato determine whether the other car is within a turn lane).

The computer 110 may also access data 134 relating to certain types ofobjects that the vehicle 101 may encounter. As described above, thesensors of vehicle 101 may be used to identify, track and predict themovements of pedestrians, bicycles, vehicles, or other objects in oraround the roadway. These objects may have particular behavior patternsthat depend on the nature of the object. For example, a bicycle islikely to react differently than a tractor-trailer in a number of ways.Specifically, a bicycle is more likely to make erratic movements whencompared with a tractor-trailer. Accordingly, in predicting an objectsbehavior, computer 110 may access object data 137 that contains numerousobject classifications, such as pedestrians, bicycles, cars,tractor-trailers, etc. For each classification, the object data 137 mayalso contain behavior information that indicates how an object having aparticular classification is likely to behave in a given situation.Vehicle 101 may then autonomously respond to the object based, in part,on the predicted behavior.

In addition to classifying the object, vehicle 101 may track a currentstate of the object. The object's state may include information used todetermine the object's classification, but any type of environmental orcontextual information may be used to determine the object's currentstate, such as the object's speed, route the object has traveled, natureof the roadway on which the object is traveling, any lane changes madeby the object, or the object's use of headlights or blinkers.

For example, FIGS. 5A and 5B depict vehicle 101 as driving along side atractor-trailer 510. As provided in FIG. 5B, the tractor-trailer 510 iswithin range of several sensor fields of vehicle 101. Using the datacollected from the sensors of vehicle 101, computer 110 of FIG. 1 mayidentify object 510 as being a tractor-trailer. In addition, computer110 may determine the state of tractor-trailer 510. For example, datacollected from the sensors of vehicle 101 may be used to determine thatthe tractor-trailer is traveling at 55 miles per hour in the left-handlane of the interstate.

Once the computer 110 has determined the tractor-trailer'sclassification and state information, it may then access behaviorinformation for objects having the same, or similar, classifications andstates. Behavior information is a collection of data relating to howvarious objects act in particular contexts. For example, all of vehiclestraveling along a particular roadway over period of time could beobserved and their movements tracked. These tracked movements could thenbe stored and a model may be created that indicates how future vehiclesmay act when traveling along that particular roadway. Upon storing thetracked movement, the vehicle's classification could be stored alongwith it. In this way, the modeled behavior may not only relate to allobjects generally, but may also relate to a specific classification ofvehicle. For example, the behavior information could indicate that allvehicles entering a highway via a particular on-ramp do not usually takethe next available exit from the highway. In addition, the trackedmovements of the vehicles could indicate that the vehicles classified astractor-trailers are much more likely to change over to the right-handlane, or to take a particular highway exit than cars. If the upcomingexit is a weighing station, the tractor-trailers might only take thatexit during particular days of the week or during a particular time ofday. This information could also be accessed in connection with thetracked vehicle movements. Accordingly, once data has been collected forthe traveling along a particular route, predictions can be made as towhat a particular vehicle will do in the future.

The collection of data for vehicle or pedestrian movements may beaccomplished in any number of ways. For example, the movement ofvehicles may be tracked using satellite imagery, roadside cameras, onboard GPS data, or via sensor data acquired from vehicles similar tovehicle 101. Preferably, the behavior model will be based on a largenumber of tracked objects for each classification of object. In thisway, an accurate behavior model can be created for each classificationof objects.

Flow diagram 600 of FIG. 6 provides an example by which vehicle 101 maybe autonomously controlled in response to predicted behaviors ofsurrounding objects. Autonomous vehicle 101 may transport itself,passengers, and/or cargo between two locations by following a route. Forexample, a driver may input a destination and activate an autonomousmode of the vehicle. In response, the vehicle's computer 110 maycalculate a route using a map, its current location, and thedestination. Based on the route (or as part of the route generation),the vehicle may determine a control strategy for controlling the vehiclealong the route to the destination. For example, the control strategymay include where to turn, at what speeds to travel, what lane to travelin, where to look for traffic signals, where to stop for intersectionsor stop signs, etc. Vehicle 101 implements the determined controlstrategy by traveling to travel along the route (Block 610). Whiletraveling in accordance with the control strategy, vehicle 101 maydetect the presence of an object within one or more of the vehicle'ssensor fields (Block 620). Upon detecting the object, the vehicle'scomputer 110 may classify the object based on the data received by thevehicle's sensors (Block 630). For example, the sensor data could beused to classify objects as being a pedestrian, bicycle, sports car,pick-up truck, etc. As described above, the vehicle's computer 110 alsouses the sensor data to determine the object's state, such as speed andlane position. (Block 640). Upon determining the objects classificationand state, computer 110 accesses behavior model information contained indatabase 137 (Block 650). Specifically, the computer 110 may accesslikely behavior patterns for objects having a similar classification andstate. Based on the behavior model data, the computer 110 may implementa new or supplemental control strategy, for example, by keeping agreater distance from a certain classification of vehicles when they aretraveling in particular lanes of traffic or at some particular speed(Block 660). The computer 110 may use the sensor data to continue totrack the object's proximity to vehicle 101 (Block 670). If the objectremains within a predefined distance of vehicle 101, such as between 0to 200 meters, then computer 110 may maintain the new control strategy(Block 680). The new control strategy may be adjusted, however, as bothvehicle 101 and the detected object change states (e.g., change speed orlocation). If the object is no longer in proximity to vehicle 101,computer 110 may return to the original control strategy, makingadjustments based on other changes to the vehicle's state (Block 690).

By implementing aspects of flow diagram 600, vehicle 101 will be able toautonomously react to surrounding vehicles or pedestrians in a way thatminimizes the risk of accidents or other unwanted events. For example,as described above, vehicle 101 may be autonomously driving along aparticular section of highway and may classify vehicle 510 as atractor-trailer that is traveling in the left-hand lane at 55 miles perhour. In implementing flow diagram 600, vehicle 101 may access thebehavior model for tractor-trailers traveling in the left-hand lane atthat section of highway. The behavior model information may indicatethat most tractor-trailers change to the right-hand lane while travelingalong that section of the highway. Accordingly, there is a highprobability of the tractor-trailer changing to the right-hand lane inthe near future. Vehicle 101 could then implement a new control strategyin response to the accessed behavior model. For example, the new controlstrategy could change the speed of vehicle 101, so as to avoid beingdirectly to the right of tractor-trailer 510. Alternatively, the newcontrol strategy could cause the vehicle 101 to change lanes or to staya particular distance from the tractor-trailer. In this way, the risk ofthe tractor-trailer 510 inadvertently making contact with vehicle 101will be reduced. The new control strategy is therefore specific to thepredicted behavior of tractor-trailer 510, in that a different behaviormodel and a different control strategy would be used if object 510 was amotorcycle or sports car.

Vehicle 101 may include one or more user input devices that enable auser to provide information to the autonomous driving computer 110. Forexample, a user, such as passenger 290, may input a destination (e.g.,123 Oak Street) into the navigation system using touch screen 217 orbutton inputs 219. In another example, a user may input a destination byidentifying the destination. In that regard, the computer system mayextract the destination from a user's spoken command (e.g., by statingor inputting “De young museum” as in the example of FIGS. 2 and 3).

The vehicle may also have various user input devices for activating ordeactivating one or more autonomous driving modes. In some examples, thedriver may take control of the vehicle from the computer system byturning the steering wheel, pressing the acceleration or decelerationpedals. The vehicle may further include a large emergency button thatdiscontinues all or nearly all of the computer's decision-making controlrelating to the car's velocity or direction. In another example, thevehicle's shift knob may be used to activate, adjust, or deactivatethese autonomous modes.

Computer 110 may include, or be capable of receiving information from,one or more touch sensitive input apparatuses 140. For example, computer110 may receive input from a user input apparatus and use thisinformation to determine whether a passenger is in contact with, such asby holding or bumping, a particular portion of vehicle 110. The touchsensitive input apparatuses may be any touch sensitive input devicecapable of identifying a force, for example a force resistance tape maybe calibrated to accept or identify a threshold pressure input (such as10 grams of pressure) or a range of pressures (such as 5-20 grams ofpressure).

Again, these inputs may be understood by the computer as commands by theuser to, for example, enter into or exit from one or more autonomousdriving modes. For example, if the vehicle is being operated in anautonomous mode and the driver bumps the steering wheel, if the force isabove the threshold input, the vehicle may go from an autonomous mode toa semi-autonomous mode where the driver has control of at least thesteering.

The various systems described above may be used by the computer tooperate the vehicle and maneuver from one location to another. Forexample, a user may enter destination information into the navigation,either manually or audibly. The vehicle may determine its location to afew inches based on a combination of the GPS receiver data, the sensordata, as well as the detailed map information. In response, thenavigation system may generate a route between the present location ofthe vehicle and the destination.

When the driver is ready to relinquish some level of control to theautonomous driving computer, the user may arm the computer. The computermay be armed, for example, by pressing a button or by manipulating alever such as gear shifter 220. Rather than taking control immediately,the computer may scan the surroundings and determine whether there areany obstacles or objects in the immediate vicinity which may prohibit orreduce the ability of the vehicle to avoid a collision. In this regard,the computer may require that the driver continue controlling thevehicle manually or with some level of control (such as the steering oracceleration) before entering into a fully autonomous mode.

Once the vehicle is able to maneuver safely without the assistance ofthe driver, the vehicle may become fully autonomous and continue to thedestination. It will be understood that the driver may continue toassist the vehicle by controlling, for example, steering or whether thevehicle changes lanes, or the driver may take control of the vehicleimmediately in the event of an emergency.

The vehicle may continuously use the sensor data to identify objects,such as traffic signals, people, other vehicles, and other objects, inorder to maneuver the vehicle to the destination and reduce thelikelihood of a collision. The vehicle may use the map data to determinewhere traffic signals or other objects should appear and take actions,for example, by signaling turns or changing lanes.

Once the vehicle has arrived at the destination, the vehicle may provideaudible or visual cues to the driver. For example, by displaying “Youhave arrived” on one or more of the electronic displays.

In one aspect, the features described above may be used in combinationwith larger vehicles such as trucks, tractor trailers, or passengerbusses. For such vehicles, the system may consider additionalinformation when computing how to control the vehicle safely. Forexample, the physical attributes of a tractor trailer, such as itsarticulation and changing weight, may cause it to maneuver verydifferently than smaller passenger cars. Larger vehicles may requirewider turns or different levels of acceleration and braking in order toavoid collisions and maneuver safely. The computer may consider thegeometry of the vehicle when calculating and executing maneuvers such aslane changes or evasive actions.

The vehicle may be only partially autonomous. For example, the drivermay select to control one or more of the following: steering,acceleration, braking, and emergency braking.

The vehicle may also address driver impairment. For example, if a driverhas been unresponsive, has reduced cognitive abilities, or has beendetected as having fallen asleep, the vehicle may attempt to wake orotherwise prompt the driver to respond. By way of example only, a cameracapturing the driver's face may be used to determine whether thedriver's eyes have remained closed for an extended period of time. Ifthe driver remains unresponsive, the computer may cause the vehicleslow, stop or pull over to a safe location, or may assume control overthe vehicle's direction or speed to avoid a collision.

In another example, the system may be always on with the driverprimarily in control of the vehicle, but only intervene and take actionwhen the vehicle detects that there is an emergency situation. Forexample, if a thick fog reduces the visibility of the driver, thevehicle may use its sensors to detect the presence of objects in thevehicle's path. If the vehicle determines that a collision is imminentyet the driver has taken no action to avoid the collision, the vehiclemay provide the driver with a warning and, absent further correction bythe driver, take one or more evasive actions such as slowing, stoppingor turning the vehicle.

The vehicle may also improve driver performance while the vehicle isunder the control of a driver. For example, if a driver fails tomaintain the vehicle's position within a lane without using a turnsignal, the vehicle may slightly adjust the steering in order to smoothout the vehicle's movements and maintain the vehicle's position withinthe lane. Thus, the vehicle may mitigate the likelihood of undesirableswerving and erratic driving. In another embodiment, if a driver is on acourse to change lanes but has not activated the turn signal, thevehicle may automatically activate the turn signal based on the detectedcourse of the vehicle.

The vehicle may require a driver to identify him or herself beforebeginning operation of the vehicle or taking some action. For example,the user may have to enter a user name and password upon entering thevehicle or use an RFID badge or other item to automatically provide hisor her user information. In addition or alternatively, the user mayidentify a class of drivers to which the user belongs. Once the vehiclehas identified the driver, the vehicle may enter into a mode that isspecific to that driver or class of driver.

Each driver of a vehicle may have his or her own driver profile.Profiles may describe the level of acceleration or impose otherlimitations on the user's operation of the vehicle. This allows forvariable degrees of operation, or types of system functions, dependingon the driver. Similarly, passengers may also have profiles. Forexample, once a passenger has been identified as described above, thevehicle may operate according to the passenger profile. The vehicle mayfurther operate differently when the vehicle is full of children, orloaded (or overloaded) with particular cargo.

Restrictions may be based on the age or experience of the vehicle'soperator. For example, a sixteen-year-old operator may havesubstantially more restrictions than a thirty-year-old experiencedoperator. In one example, a young driver may be subject to certainrestrictions including limits on speed or locations regarding where theuser may travel, how fast the vehicle may approach an intersection, orwhere the vehicle is located between lane lines or boundaries. Yetfurther, a new driver's freedom to operate the vehicle at may decreaseat night, when the vehicle is near schools, about to enter high trafficareas, or otherwise enter a territory that may strain the experience ofa new driver. Similarly, the vehicle may also be used to control themovements of drivers subject to legal restraints. For example, thoseconvicted of certain crimes, such as endangering the welfare of a child,may be prevented from using the vehicle within a certain distance from aschool or playground.

If the vehicle approaches an area restricted to the driver, the car mayslow to a stop or prevent the driver from turning the vehicle into anarea. Similarly, if the driver takes the vehicle into a restricted area,the vehicle may take control of speed and direction of the vehicle tomaneuver away from the restricted area.

The vehicle may also allow certain drivers greater privileges withrespect to operating the vehicle at fast speeds or taking other certainactions.

The identification system may be used to not only identify the driver,but also to identify the purpose of the drive. For example, aschoolteacher or other figure may use a badge to identify himself orherself to the computer before placing a child in the vehicle. If thebadge or other user is not recognized, the vehicle may not permit theteacher to place the child in the vehicle, such as by not opening thedoors. After the child has been placed in the vehicle, the vehicle mayenter into a fully autonomous mode or one where the child may havelimited control in order to transport the child to a destination. Oncethe vehicle has arrived at its destination, for example, the child'shome, the vehicle may only unlock the doors when the parent, guardian,or other authorized user has submitted a badge or identified himself orherself to the vehicle.

A similar system may also be used to transport packages or other items.For example, the courier service may use the badge to open the vehicle,place a package in the vehicle, and input a destination and theidentification of the recipient. In response, the vehicle may enter afully autonomous mode and transport the package to the destination. Oncethe vehicle has arrived, it may wait for the recipient to identify himor herself before opening and relinquishing the package. In anotherexample, the vehicle may be sent to pick up food from a restaurant orgroceries from a market in a similar manner.

The vehicle may also be used to train and test drivers. For example, thevehicle may be used to instruct new drivers in the real world, but maycontrol various aspects of the ride in order to protect the new driver,people external to the car, and external objects. In another example,new drivers may be required to control the vehicle safely for someperiod. The vehicle may then determine whether the user is a safe driverand thus determine whether the user is ready to be a licensed driver.

The vehicle may also have additional user interfaces designed oraugmented for safety. For example, the steering wheel may be configuredto be stiffer when the steering wheel or computer detects that thedriver is moving the vehicle into another lane without a turn signal.This may prevent sudden jerks or movements of the vehicle. Driversgenerally use small movements of the steering wheel to maintain positionwithin a lane, and thus the computer may disregard these movements andmaintain the position with the lane independently from the driver'ssteering. Similarly, moving the wheel somewhat harder to the left maysignal a lane change to the computer. If the computer determines thatthere are no obstacles and the roadway allows for this movement, thevehicle may respond to the hard turn of the steering wheel by changinglanes. In that regard, turning the steering wheel may not indicate theorientation of the front wheels, but rather the user's desire to take acertain action. In response, the computer may handle the requirementsassociated with moving the vehicle into another lane.

The acceleration or deceleration pedals may also be reconfigured toincrease safety. For example, if the vehicle has identified the speedlimit of the roadway, the driver's full depression of the acceleratormay be interpreted as an indication that the driver wants to drive atthe posted speed limit or some other pre-determined limit. It may alsoindicate that the car should navigate turns at the highest speed that issafest for the turn. Yet further, a young driver may be limited to 20miles per hour where the posted speed limit is 25 miles per hour or 15miles per hour in an emergency construction zone. In this regard, thedriver's effect on the acceleration and deceleration of the vehicle maybe controlled based on the computer's identification of who is drivingand the objects or obstacles that are within the surroundingenvironment.

In another example, the vehicle may use the windshield or an electronicdisplay to display information unrelated to the vehicle, such as email,during a fully autonomous mode. If the computer determines that thedriver's assistance is needed, the vehicle may turn the display off andflash a warning to the driver such as “you need to pay attention rightnow in case I need you.” For example, the information unrelated todriving may be turned off if roadway information becomes unavailable, orif traffic has been reconfigured around an accident.

The vehicle may also park itself. For example, the map information mayinclude data describing the location of parking spots along a roadway orin a parking lot. The computer may also be configured to use its sensorsto determine potential parking spots, such as causing the vehicle totravel down a road and checking for painted lines along a street thatindicate an open parking space. If computer determines another vehicleor object is not within the spot, the computer may maneuver the vehicleinto the parking spot by controlling the steering and speed of thevehicle. Using the method described above, the vehicle may also classifyany objects that are near the potential parking spot, and position thevehicle within the parking spot based on those surrounding objects. Forexample, the vehicle may position itself closer to an adjacent bicyclethan it would an adjacent truck.

The vehicle may also have one or more user interfaces that allow thedriver to reflect the driver's driving a style. For example, the vehiclemay include a dial which controls the level of risk or aggressivenesswith which a driver would like the computer to use when controlling thevehicle. For example, a more aggressive driver may want to change lanesmore often to pass cars, drive in the left lane on a highway, maneuverthe vehicle closer to the surrounding vehicles, and drive faster thanless aggressive drivers. A less aggressive driver may prefer for thevehicle to take more conservative actions, such as somewhat at or belowthe speed limit, avoiding congested highways, or avoiding populatedareas in order to increase the level of safety. By manipulating thedial, the thresholds used by the computer to calculate whether to passanother car, drive closer to other vehicles, increase speed and the likemay change. In other words, changing the dial may affect a number ofdifferent settings used by the computer during its decision makingprocesses. A driver may also be permitted, via the user interface 225,to change individual settings that relate to the driver's preferences.In one embodiment, insurance rates for the driver or vehicle may bebased on the style of the driving selected by the driver.

Aggressiveness settings may also be modified to reflect the type ofvehicle and its passengers and cargo. For example, if an autonomoustruck is transporting dangerous cargo (e.g., chemicals or flammableliquids), its aggressiveness settings may be less aggressive than a carcarrying a single driver—even if the aggressive dials of both such atruck and car are set to “high.” Moreover, trucks traveling across longdistances over narrow, unpaved, rugged or icy terrain or vehicles may beplaced in a more conservative mode in order reduce the likelihood of acollision or other incident.

In another example, the vehicle may include sport and non-sport modeswhich the user may select or deselect in order to change theaggressiveness of the ride. By way of example, while in “sport mode”,the vehicle may navigate through turns at the maximum speed that issafe, whereas in “non-sport mode”, the vehicle may navigate throughturns at the maximum speed which results in g-forces that are relativelyimperceptible by the passengers in the car.

The vehicle's characteristics may also be adjusted based on whether thedriver or the computer is in control of the vehicle. For example, when aperson is driving manually the suspension may be made fairly stiff sothat the person may “feel” the road and thus drive more responsively orcomfortably, while, when the computer is driving, the suspension may bemade such softer so as to save energy and make for a more comfortableride for passengers.

The driver may also train the vehicle to the driver's specific style.For example, the vehicle may include a “record” button to put thevehicle into a training mode to record the actions of the driver. Thedriver may drive the vehicle for a day while the vehicle monitors howmuch torque is applied to the steering wheel, how quickly the driveraccelerates at an interaction or highway, whether the driver selects tochange lanes and pass slower vehicles, and how the driver applies thebrakes. The vehicle may then identify the driver's preferred style andreplicate this style when the driver uses the vehicle in an autonomousmode.

The record button may also be used to record a specific route that thedriver follows each day as well as the driver's style during the route.Then, the driver may select a “play” button and replay the route on theuser display. Alternatively, the driver may select a “repeat trip”button, causing the vehicle to follow the same route making similarchoices, though making alterations as necessary for safety, as thedriver had done during the recording mode.

The vehicle may include a sleeping mode that allows the driver to givefull control of the vehicle to the computer so that the driver may sleepor reduce his or her focus on the roadway. For example, the vehicle mayinclude a user input device that allows the user to input informationsuch as the duration of the sleep mode, e.g., 20 minutes, 4 hours, 8hours, etc. In response, the vehicle may drive slower or on lesstraveled roadways, select a route that will get the driver to thedestination in the identified period, or select a route which that avoidbumps or other disturbances to the driver.

If the duration of the sleep mode is greater than the estimated amountof time that it will take for the vehicle to arrive at the destination,the vehicle may park itself at some point along the route in order toarrive at the destination at the end of sleep mode. Alternatively, thevehicle may proceed to the destination without stopping, and use audibleor tactile cues to wake the driver after the identified period hadpassed. Such cues may also be activated if the duration of the sleepmode ended prior to the vehicle arriving at the destination.

The driver may also select to have his or her vehicle communicate withother devices. As shown in FIG. 7, vehicle 101 may communicate over anetwork 720 with devices such as a remote server 710, a personalcomputer 730, a mobile device 740, or another autonomous vehicle 702. Inaddition, vehicles, such as vehicle 701 and vehicle 702, may wirelesslytransmit information directly to nearby vehicles using radio, cellular,optical or other wireless signals. Alternatively, vehicles maycommunicate with each via nodes that are shared among multiple vehicles,e.g., by using cell towers to call other cars or transmit and sendinformation to other cars via the Internet.

The transmitted information between vehicles may include, for example,data describing the vehicle or the vehicle's environment.

In one example, a driver of a first vehicle may select an option toallow other vehicles on the roadway to transmit information from thevehicle's sensors or computer. This information may include detailsabout the first vehicle's environment such as detected objects, trafficconditions, or construction. The information transmitted to othervehicles may be sensor data unprocessed by the first computer orinformation previously processed by the first computer in order toreduce the time needed to obtain and process the information at a secondvehicle. If the second autonomous vehicle is behind the first vehicle,it may use the information to determine how to maneuver the vehicle. Byway of example, if the first vehicle is only a few car lengths in frontof the second vehicle and it detects a moving object, the first vehiclemay transmit information relating to the moving object to the secondvehicle. If the second vehicle determines that the object is movingtowards the second vehicle's path, the second vehicle may slow down. Yetfurther, if the second vehicle is a few miles behind the first vehicleand the first vehicle determines that it is in a traffic jam (e.g., bydetermining that its speed is substantially less than the road's speedlimit), the second vehicle may select an alternate route.

The driver of the first vehicle may also select to share informationregarding the operation of the vehicle to other vehicles. For example, asecond vehicle may also receive information indicating that the firstvehicle's traction control has been activated around a turn and inresponse, the second vehicle may take the same turn at a slower speed.In another example, if the first vehicle has taken an evasive action,such as swerving to avoid a deer, a second vehicle may react by slowingdown or taking some other evasive action to avoid the likelihood of acollision.

In addition to sharing information, the driver may also select to havethe vehicle act cooperatively with other autonomous vehicles. Thecooperation may be controlled at least in part by the vehicle'scollective computers, a single one of the cooperating computers, or by aremote server to which the cooperating computers relinquish control.

For example, two or more cooperating vehicles may travel together in aline such that one vehicle follows closely behind the others for variousdistances sharing information and synchronizing speeds so that thefollowing vehicle may draft behind the leading vehicle; such actions maypreserve fuel and increase the flow of traffic. The cooperating vehiclesmay travel one in front of the other, as a line of two or more vehicles.The first vehicle in a line may reduce the amount of wind resistance onthe other the vehicles in the line and may increase the fuel efficiencyof the other vehicles. The first vehicle may be rotated, for example byexiting the first vehicle from the line, slowing the vehicle down, andreentering at the end of the line. This may allow for greaterdistribution of the fuel costs along the line. The line may allow alarge number of vehicles to travel at fast speeds with only a few feetor inches between the vehicles, thus reducing the amount of space neededfor a line of vehicles and increasing the flow of traffic. Thiscooperating may also be used to increase the flow of traffic by reducingthe amount of sporadic braking by drivers during heavy trafficconditions.

For example, a vehicle approaching a line may transmit a signal to thevehicles of the line indicating the approaching vehicle's intention tojoin the line. If the approaching vehicle is alongside the line, some ofthe vehicles may slow down slightly to open a gap in the line. Theapproaching vehicle may join the line, and the vehicles may speed upslightly to close the gap. When a vehicle is ready to leave the line,for example to use an exit, a second signal is transmitted, and the lineresponds by opening a gap around the leaving vehicle. Once the leavingvehicle has left the line, the line may again close the gap.

The line described above may be a service offered to drivers for a fee.For example, the driver of a vehicle wishing to join the line may berequired to pay the other drivers or the train or some other servicebefore joining.

In addition to cooperatively driving together in lines, autonomousvehicles may also communicate in order to increase convenience andsafety on the roadways. For example, autonomous vehicles may be able todouble (two vehicles in a row) and triple park (three vehicles in a row)next to other autonomous vehicles. When a driver would like to use avehicle which is parked in or surrounded by other autonomous vehicles,the driver's vehicle may send a signal instruction the other vehicles tomove out of the way. The vehicles may respond by cooperativelymaneuvering to another location in order to allow the driver's vehicleto exit and may return to park again.

In another example, the cooperation mode may be used to promote smarterroad crossings. For example, if several autonomous vehicles areapproaching and intersection, the right-of-way problem, or which vehicleshould be next to enter the intersection, may be calculated anddetermined cooperatively among the several vehicles. In another example,traffic signals may change quickly, such as within only a few seconds orless, to allow more vehicles to pass through an intersection in multipledirections. The vehicle may only need the traffic signal to be green fora second or less in order to pass through the intersection at highspeeds.

Vehicle 101 may also receive updated map or object data via network 720.For example, server 710 may provide vehicle 101 with new data relatingto object classifications and behavior model information. Computersystem 110, of FIG. 1, may then be updated, and the new data may be usedin controlling the vehicle autonomously, such as through implementationof flow diagram 600.

In addition to the sensors described above, the vehicle may also includevarious other sensors in order to increase the perceptive abilities ofthe vehicle. For example, thermal imaging sensors may be used toidentify the heat of pedestrians, animals, or vehicles. In anotherexample, detecting the rigidity of certain objects is generallydifficult, however using near infra-red sensors may be used to detect togive the computer additional cues. Sensors may also be used to detectthe conditions of the road surface, such as icy or wet. These sensorsmay be included, for example, in the tires or wheels. Wind resistancesensors, placed for example on each of the lateral sides of the vehicle,may be used to increase fuel efficiency by allowing the computer toadjust the course of the vehicle.

Sensors may also be incorporated into the interior of the vehicle inorder to sense information about any occupants. Sensors may be used toidentify the state of the driver's eyes, breathing, temperature, orheart rate to determine whether the driver is sleeping, temporarilydistracted, etc. Using the data collected from these sensors, thevehicle may change its operation. For example, the vehicle may determinethat the driver has fallen asleep or passed out and take control inorder to avoid a collision such as by slowing or parking the vehicle. Inother examples, breath sensors or sensors which detect eye movements maybe used to determine whether the driver is intoxicated, and if so, thevehicle may prevent the driver from taking control but not from usingthe vehicle to drive home or to a hospital.

As these number and usage of these autonomous vehicles increases,various sensors and features may be incorporated into the environment toincrease the perception of the vehicle. For example, low-cost beacontransmitters may be placed on road signs, traffic signals, roads orother highway infrastructure components in order to improve thecomputer's ability to recognize these objects, their meaning, and state.Similarly, these features may also be used to provide additionalinformation to the vehicle and driver such as, whether the driver isapproaching a school or construction zone. In another example, magnets,RFID tags or other such items may be placed in the roadway to delineatethe location of lanes, to identify the ground speed vehicle, or increasethe accuracy of the computer's location determination of the vehicle.

In other examples, autonomous vehicles may be given certain privilegeson the roadway. For example, special lanes may be designated forautonomous vehicles traveling in lines so that they may travel at fasterspeeds. In another example, certain geographic areas may be designatedas autonomous vehicle-only zones.

These vehicles may also be used to assist law enforcement. For example,an autonomous vehicle may be controlled by a remote driver in hostilesituations or areas in order to reduce the risk of injury or death tothe driver. In another example, autonomous vehicles may be used toassist law enforcement officers while on patrol by controlling thevehicle in order to allow the officer to focus on looking around,running license plates, or using the radio rather than driving. Apatrolling vehicle may also have a pre-determined route in order toincrease the efficiency of patrolling. If needed, the driver may takecontrol in order to take the vehicle off of the route. The patrollingvehicle may also inform the driver if any areas of the pre-determinedroute have been skipped. In addition, the driver may also receiveprocessed sensor data from the computer in order to increase thedriver's perception of the environment.

These vehicles may also be used to assist law enforcement with dangerousdrivers. During dangerous high-speed chases, the vehicle may allowautonomous law enforcement vehicles to maintain high speeds in confinedsituations, such as through areas of high traffic, while at the sametime reducing the likelihood of a collision with environmental objects.In another example, autonomous law enforcement vehicles or otherautonomous vehicles may voluntarily block in a dangerous driver oradjust the vehicles' formation to prevent dangerous maneuvers until lawenforcement vehicles arrive. Autonomous vehicles may also keeppassengers away from other humans or vehicles (autonomous or not)behaving dangerously, for example, a swerving or speeding vehicle ordistracted a driver talking on a mobile phone.

In case of an emergency, autonomous law enforcement vehicles such aspatrol vehicles, fire trucks, or ambulances may be given a limitedamount of control over nearby vehicles. For example, these autonomousemergency vehicles may transmit a signal causing other autonomousvehicles to turn off certain features or maneuver out of the way inorder to allow the emergency vehicle to pass safely. In another example,law enforcement may be able to take control of a vehicle at aconstruction zone or accident scene in order to send the vehicle throughquickly while again reducing the likelihood of a collision. Emergencyvehicles may also be able to control the directionality of a lane, suchas by temporarily changing the direction of a one-way street or changinga right-turn-only lane to a lane through an intersection.

Autonomous vehicles may also be controlled remotely. For example, if thedriver is asleep, the sensor data may be sent to a third party so thatvehicle may continue to have a responsive operator. While delay andlatency may make this type of telemetry driving difficult, it may forexample be used in emergency situations or where the vehicle has gottenitself stuck. The vehicle may send data and images to a central officeand allow a third party to remotely drive the vehicle for a short perioduntil the emergency has passed or the vehicle is no longer stuck.

These vehicles may also be subject to zoning rules. For example, wherethe vehicle enters a pre-designated “autonomous zone,” such as anairport, an airport control service may take control of the vehicle,deliver it to a specified loading or unloading location, and park thevehicle. This may allow the airport service controlling the vehicle tolimit the number of vehicles in a given area, such as a passengerloading or unloading zone at an airport, or to decrease the likelihoodof a collision in a high trafficked area. In another example, some zonesmay require that the driver take complete control of the vehicle. Acomplicated traffic circle may send a signal indicating that vehiclescannot be driving in an autonomous mode and thus require that the drivertake control.

Different zones may also have different access levels. For example, asmentioned above, younger drivers may be prohibited from accessingcertain areas. Other zones may be prohibited to all but law enforcementofficers or employees of a business. In another example, zones may alsohave speed or vehicle occupancy (carpooling) requirements.

Autonomous vehicles may be configured differently than currentnon-autonomous vehicles. For example, current vehicles may beconsiderably heavier than necessary in order to increase theeffectiveness of safety features or may have stiffer suspensions andlarger engines to increase performance and handling. Autonomous vehiclesmay be built to be smaller and lighter with softer suspensions andsmaller engines. This may allow autonomous vehicles to be much morecompact and efficient in terms of fuel economy than non-autonomousvehicles.

These vehicles may also have various seating and control arrangements.For example, the vehicle may include a sleeping surface to allow thedriver to lie down and sleep while the vehicle is in control. Theseating within the vehicle may be arranged so that the driver andpassengers may face one another. Televisions or web browsers, and othercustomer media may be incorporated in order to increase availableactivities and passenger comfort levels. The vehicle controls may bedistributed throughout the vehicle so that each passenger may controlthe vehicle if necessary. The distributed controls may require that thedriver have the ability to have complete control, but the remainingpassengers may only be able to steer, slow or stop the vehicle. Thedistributed controls may also have a locking mode in order to preventcertain passengers from having any control over the vehicle.

The vehicle may also be made to change shape based on its use. Forexample, when not in use or when in a parking mode which requires thevehicle to continue driving, the vehicle may collapse (e.g., the roofmay fold down). This may reduce the volume of the vehicle and allowother drivers to see past or over the vehicle. The reduced volume mayalso allow the vehicle to be more portable and efficient.

The vehicle may also drive itself to optimize its longevity and reducewear of the various components. For example, the vehicle may operate thebrakes gradually whenever able in order to reduce wear on the tires andbrakes or accelerate slowly in order to reduce wear on the tires. Avehicle may also adjust suspension, torque or horsepower settings basedon the conditions of the roadway (icy, wet, smooth, rocky or bumpy, lotsof potholes, etc.). In another example, electric autonomous vehicles maydrive themselves to a recharging station to charge one or more batterieswhen not in use by a human operator, or fueled vehicles may drivethemselves to refueling stations on their own. Vehicles with trailersmay detect and correct for trailer oscillations and changes in theweight or maneuverability of a trailer in order to increase performanceand smoothness of the ride.

The vehicle may also diagnose problems and take one or more actions inorder to prevent further damage or remedy any problems. For example, thevehicle may monitor the performance of various components such as theengine, brakes or tires. If there is an emergency, such as if coolant isleaking and the engine is at risk of overheating, the computer may stopthe vehicle immediately to avoid any further damage. If there appears tobe a less imminent problem, such as a tire slowly leaking or thevehicle's components are worn to some pre-determined level, the vehiclemay schedule and transport itself to the manufacturer or a repair shopfor maintenance. In one example, the vehicle may determine that the oilneeds to be changed, schedule an appointment, and take itself for theoil change. In another example, the vehicle may test itself by drivingshort or large distances in order to diagnose problems. Vehicles mayalso detect and self-correct minor problems such as incorrect wheelalignment or tire pressure. A fleet vehicle may diagnose and reportproblems or variations among the other vehicles of the fleet.

These vehicles may produce data which may benefit non-drivers. Forexample, advertisers or businesses may pay a fee for informationregarding how often vehicles arrive at a business location or how oftenvehicles make u-turns before arriving at the business location. Afueling station operator may want to know how many vehicles drive by alocation during a particular period, the fuel levels of the vehicles,and decide whether the location would be a profitable location for afueling station.

As these and other variations and combinations of the features discussedabove can be utilized without departing from the invention as defined bythe claims, the foregoing description of exemplary embodiments should betaken by way of illustration rather than by way of limitation of theinvention as defined by the claims. It will also be understood that theprovision of examples of the invention (as well as clauses phrased as“such as,” “e.g.”, “including” and the like) should not be interpretedas limiting the invention to the specific examples; rather, the examplesare intended to illustrate only some of many possible aspects.

The invention claimed is:
 1. A method comprising: controlling a vehicle according to a first control strategy, the first control strategy determined based on a route to a destination; detecting an object external to the vehicle using one or more sensors; using a processor, determining a classification and a state of the detected object; determining a current location of the vehicle; accessing behavior data for other objects having the same or similar classifications and states as the detected object, the behavior data indicating how the other objects have operated at the current location at one or more previous times; predicting a likely behavior of the detected object at the current location based on the behavior data for the other objects having the same or similar classification and state as the detected object; and modifying the first control strategy to obtain a second control strategy for controlling the vehicle based on the predicted likely behavior of the detected object.
 2. The method of claim 1, wherein the classification of the detected object includes one of an automobile, a pedestrian, structure, or a bicycle.
 3. The method of claim 1, wherein the detected object is an automobile, and wherein the classification of the detected object includes the type of automobile.
 4. The method of claim 1, wherein the determining a classification of the detected object is based on identifying at least one of a logo, a bumper sticker, or a license plate.
 5. The method of claim 1, wherein the state of the detected object relates to at least one of: location, traffic lane in which the detected object is traveling, speed, acceleration, entry onto a road, exit off of a road, activation of headlights, activation of taillights, or activation of blinkers.
 6. The method of claim 1, wherein the behavior data is based on movement data for a plurality of other objects at one or more locations.
 7. The method of claim 6, wherein the movement data are tracked using one of: satellite imagery, roadside cameras, on-board GPS data, or sensor data acquired for other nearby vehicles.
 8. The method of claim 1, wherein: implementing the second control strategy comprises providing a command to orient the vehicle in a position and velocity based at least in part on the likely behavior of the detected object; and providing the command to orient the vehicle includes positioning the vehicle at a predetermined distance from the detected object, the predetermined distance being based, at least in part, on the classification of the detected object.
 9. The method of claim 1, wherein the likely behavior of the detected object is provided as a probability of the detected object entering to one or more states.
 10. The method of claim 1, further comprising: receiving updated behavior data; and wherein predicting the likely behavior of the detected object is based at least in part on the updated behavior data.
 11. The method of claim 10, wherein receiving the updated behavior data comprises accessing a remote server.
 12. A system comprising: one or more sensors for detecting an object in a vehicle's surroundings; and a processor coupled to the one or more sensors, the processor being operable to: control operation of the vehicle based on a first control strategy, wherein the first control strategy is determined based on a route to a destination; determine a classification and a state of a detected object; determine a current location of the vehicle; access behavior data for other objects having the same or similar classifications and states similar as the detected object, the behavior data indicating how the other objects have operated at the current location at one or more previous times; predict a likely behavior of the detected object at the current location based on the behavior data for the other objects having the same or similar classification and state as the detected object; and modify the first control strategy to obtain a second control strategy for controlling the vehicle based on the predicted likely behavior of the detected object.
 13. The system of claim 12, wherein the classification of the detected object includes classifying the detected object as one of: an automobile, a pedestrian, a bicycle, or a structure.
 14. The system of claim 12, wherein the detected object is an automobile, and wherein the classification of the detected object includes identifying the type of automobile.
 15. The system of claim 12, wherein the classification of the detected object is based on at least one of: a logo, a bumper sticker, or a license plate.
 16. The system of claim 12, wherein the state of the detected object relates to at least one of: location, traffic lane in which the detected object is traveling, speed, acceleration, entry onto a road, exit off of a road, activation of headlights, activation of taillights, or activation of blinkers.
 17. The system of claim 12, wherein the behavior data represents information regarding the states of the other objects that have been tracked at one or more locations.
 18. The system of claim 17, wherein the objects have been tracked using at least one of: satellite imagery, roadside cameras, on-board GPS data, or via sensor data acquired from other nearby entities.
 19. The system of claim 12, wherein the second control strategy includes positioning the vehicle at a predetermined distance from the detected object, the predetermined distance being based, at least in part, on the classification of the detected object.
 20. The system of claim 12, wherein the likely behavior of the detected object is provided as a probability of the detected object entering to one or more states.
 21. The system of claim 12, wherein the processor is further operable to access updated behavior data; and wherein the likely behavior of the detected object is based at least in part on the updated behavior data.
 22. The system of claim 21, wherein the vehicle receives the updated behavior data from a remote server.
 23. A method for providing data in connection with controlling of a vehicle, the method comprising: storing, at a server, behavior data for a plurality of objects, wherein the behavior data includes classification and state information for at least a subset of the plurality of objects at one or more locations and at one or more times; receiving identification data from a remote device, wherein the identification data includes data relating to a detected object near the remote device; receiving a location of the detected object; determining, at the server, the classification and state of the detected object; predicting a likely behavior of the detected object based on the behavior data for the plurality of objects having a classification and state similar to the detected object for the location where the detected object was detected and the time when the detected object was detected; transmitting the likely behavior of the detected object to the remote device. 